The invention relates to electronic devices, and, more particularly, to color cameras and methods.
Color video cameras can use various type of color image detectors. Single-detector color image detectors avoid the optical complexity and problems with image registration of multiple detector setups, such as with three detectors: one for each of the primary colors. However, the single detector color imaging has difficulties because at least three distinct types of color information must be extracted from the single detector. Known approaches include using a color wheel in front of the single detector to time modulate the frequency bands sensed. More commonly, a color filter array in front of the single detector permits a pixel to be illuminated by only a selected band of frequencies. Examples of color filter arrays include (1) repeating interlaid patterns of one luminance filter and two chrominance filters as illustrated in FIGS. 1A-B of U.S. Pat. No. 3,971,065 and (2) parallel vertical red, green, and blue alternating striped color filters superimposed on the single image detector which is scanned horizontally. The single detector in these setups may be charge coupled devices (CCDs) or other such devices.
In particular, FIGS. 1a-c heuristically shows a camera having a CCD array detector with parallel red, green, and blue stripe filters one pixel wide and running vertically. FIG. 1a shows in perspective view generic camera elements: a lens for focussing an image on the CCD as suggested by representative light rays plus control, memory, and readout electronics. The camera may be a still camera (one image created per camera activation) or a real time video camera (a continuous stream of images). FIG. 1b shows in plan view the CCD with an image area of 488 rows by 754 columns of pixels, a column output multiplexer, and three output shift registers with an output buffer. FIG. 1c illustrates a portion of the CCD image area showing columns of pixels between channel stops and with a red, green, or blue stripe filter over each column. The CCD operates as follows: first during an integration time period the clocked gates are biased high and photons passing through a filter and a clocked gate are absorbed in a potential well in the underlying silicon (forming one pixel) and generate electron-hole pairs with the electrons trapped in the potential well. Next, during a readout time period each packet of trapped electrons is successively transferred from its original potential well to an adjacent potential well under a virtual gate (solid arrows), and then from the potential well under the virtual gate to the next potential well under a clocked gate (broken arrows) by toggling the clocked gates between high and low biases. At the end of each set of three columns a multiplexer transfers the charge packets from a column to the appropriate one of three output shift registers. Thus for each color channel (images of a single color) the CCD samples only every third (horizontal) pixel, and this limited sampling in the color channels diminishes the horizontal resolution and demands interpolation in each color channel for reconstruction to approximate the original full color image. However, even with interpolation, the reconstructed image loses horizontal resolution. For example, FIGS. 2a-c show, respectively, an input image which is mostly red, the three color channels as sampled (the dark columns are the small intensity green and blue portions of the image), and the reconstructed color channels using bilinear interpolation. Loss of horizontal resolution stands out.
The same need for interpolation exists in any color filter array geometry, and known camera systems typically use a non-ideal lowpass filter for the reconstruction. This non-ideal reconstruction introduces unacceptable visible artifacts into the final image such as aliasing, Moire patterns, ringing, anisotropic effects, and sample frequency ripple. To overcome these artifacts, most camera systems employ some form of optical prefiltering to limit the high spatial frequency content of an image prior to sampling. This results in a blurry image and reduced image resolution. To improve image resolution, some camera systems compute the high spatial frequency luminance from the interpolated color channels and add this to the color channels to improve high spatial frequency performance, but this still leads to poor images.
Various interpolation methods for artificial zoom and other fillings for missing pixels exist. For example, Thurnhofer et al, Adaptive interpolation of images with application to interlaced-to-progressive conversion, 2094 Proc. SPIE 614-625 (1993), uses a local gradient analysis for a directional interpolation to generate filling pixels such as in artificial zoom and interlaced field to full field. Nevertheless, such reconstruction approaches still lack the resolution required in many applications.
Many approaches exist for pattern classification. For example, decision trees based on characteristics derived from input images, such as object diameter, fill factor, and so forth, have been used for image classification. Similarly, multilayer perceptrons (or neural networks generally) learn associations by training with known classified input images. See Lippmann, "An introduction to computing with neural nets", in Neural Networks: Theoretical Foundations and Analysis (C. Lau, ed., IEEE Press, New York 1991).